Papers with crowdsourcing methods
Data Collection for Dialogue System: A Startup Perspective (N18-3)
Copied to clipboard
| Challenge: | Developing dialogue systems such as Apple Siri and Google Now requires high quality training data but data collection with crowdsourcing is largely an open question. |
| Approach: | They propose to use crowdsourcing to collect data for a user intent classification task in a dialogue system. |
| Outcome: | The proposed method improves the quality of the collected data and the model performance on real user queries. |
A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Prior attempts to develop crowdsourcing methods have either had low accuracy or required substantial expert annotation. |
| Approach: | They propose a multi-stage crowd workflow that reduces expert involvement without sacrificing accuracy. |
| Outcome: | The proposed method reduces expert effort by 4x, from 56% to 14% of cases. |
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021.acl-long)
Copied to clipboard
| Challenge: | Despite the importance of datasets for natural language understanding, there has been little attention on crowdsourcing methods for collecting datasets. |
| Approach: | They compare the effectiveness of crowdsourcing methods for boosting NLU example difficulty with training crowdworkers instead of expert judgments. |
| Outcome: | The proposed method is ineffective for boosting NLU example difficulty, but it is not effective for training crowdworkers and qualifying workers based on expert judgments. |